In today's information-rich environment, quickly handling massive volumes of data can be both challenging and important. This data may be provided in streams, with, in many instances, data values being generated in real-time, as events occur. For example, microsensors used in radio-frequency identification (RFID) in tracking and access applications can provide streaming data on locations of objects being tracked. As another example, data defining financial transactions may be provided in a stream as those transactions occur.
For many businesses the ability to operate on streaming data arriving in real-time can provide significant competitive advantage. For example, financial operations that are based on results of financial trades may receive streams of data on trades as they occur. Moreover, responding to particular signals in the streaming data quickly is often a critical aspect of many applications. As an example, network monitoring systems used by government agencies to detect security threats need to detect and report events represented in streams of data collected through monitoring.
Conventionally, processing on streaming data was performed by first storing the data in a database. The database could then be queried to retrieve the data for further processing. Therefore, analyzing the data in real-time was difficult, because of the limits imposed by database access time, particularly for streams with high data rates.
Event stream processing (ESP) is an emerging technology that enables a stream of data to be processed in real-time. Events, manifested as meaningful patterns within the data streams, are detected as a result of the stream processing. ESP provides complex event processing, which allows meaningful complex events to be detected in continuously arriving data or in a combination of stored and newly arriving data. An organization using ESP can monitor streams of data, analyze them, and respond properly to opportunities and threats.
An event processing engine constitutes a core of the event processing system and is designed to provide real-time data processing performance by eliminating the latencies characteristic of conventional approaches of storing and retrieving streams of data for later processing. In an event processing engine, the streaming data “runs through” queries, or operators, that act upon the data.
ESP technology has been applied in a commercially available stream processing platform provided by StreamBase, Inc, Lexington, Mass., U.S.A. and improved to develop a StreamBase Stream Processing Platform, the first of a new class of systems that provides the ability to process, analyze, and act on streaming information. The stream processing platform can be used to develop and execute stream processing applications that define the processing to be performed on one or more streams in any desired setting.
Despite a good performance achieved by existing stream processing platforms, improvements would be desirable.